Observations on CFD Verification and Validation 
from the AIAA Drag Prediction Workshops 


Joseph H. Morrison* and Bil Kleb t 
NASA Langley Research Center, Hampton, VA, Zip 23681 

and 

John C. Vassberg* 

The Boeing Company, Huntington Beach, CA, 92647 


The authors provide observations from the AIAA Drag Prediction Workshops that have spanned over a 
decade and from a recent validation experiment at NASA Langley. These workshops provide an assessment 
of the predictive capability of forces and moments, focused on drag, for transonic transports. It is very diffi- 
cult to manage the consistency of results in a workshop setting to perform verification and validation at the 
scientific level, but it may be sufficient to assess it at the level of practice. Observations thus far: 1) due to 
simplifications in the workshop test cases, wind tunnel data are not necessarily the “correct” results that CFD 
should match, 2) an average of core CFD data are not necessarily a better estimate of the true solution as it is 
merely an average of other solutions and has many coupled sources of variation, 3) outlier solutions should be 
investigated and understood, and 4) the DPW series does not have the systematic build up and definition on 
both the computational and experimental side that is required for detailed verification and validation. Several 
observations regarding the importance of the grid, effects of physical modeling, benefits of open forums, and 
guidance for validation experiments are discussed. The increased variation in results when predicting regions 
of flow separation and increased variation due to interaction effects, e.g., fuselage and horizontal tail, point 
out the need for validation data sets for these important flow phenomena. Experiences with a recent validation 
experiment at NASA Langley are included to provide guidance on validation experiments. 


Nomenclature 


C D , CD 

drag coefficient 

C l ,CL 

lift coefficient 

Cm. Cm, CM 

pitching moment coefficient 

c v 

coefficient of variation, a / jl 

d 

diffuser throat diameter 

h 

one-dimensional estimate of grid spacing 

lH 

horizontal tail incidence angle, degrees 

L 

length of recirculation region 

M 

Mach number 

N, NPTS 

number of solution points in a mesh 

V 

order of accuracy of numerical scheme 

Re 

Reynolds number based on reference chord 

x.y.z 

Cartesian coordinates 

Symbols 

a 

angle of attack, degrees 

A 

estimate of the population mean 

a 

estimate of the population standard deviation 
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I. Introduction 


O VER a decade ago, Computational Fluid Dynamics (CFD) became an integral tool for analysis and design in the 
aircraft industry. Since then, CFD’s role has expanded beyond the prediction of attached flow at the cruise design 
condition to more complex flow physics at off-design conditions. The AIAA CFD Drag Prediction Workshop (DPW) 
began the current series of workshops to evaluate the state of the art in CFD. DPW has evolved with the application of 
CFD to ever more complex flows of industrial interest. 

The DPW Series was initiated by a working group of members from the AIAA Applied Aerodynamics Technical 
Committee. For five workshops to date, the DPW Organizing Committee has adhered to a set of primary objectives: 

• Assess state-of-the-art CFD methods as practical aerodynamic tools for the prediction of forces and moments 
on industry-relevant geometries, with a focus on absolute drag. 

• Provide an impartial international forum for evaluating the effectiveness of CFD Navier- Stokes solvers. 

• Promote balanced participation across academia, government labs, and industry. 

• Use common public-domain subject geometries, simple enough to permit high-fidelity computations. 

• Provide baseline grids to encourage participation and help reduce variability of results. 

• Openly discuss and identify areas needing additional research and development. 

• Conduct rigorous statistical analyses of CFD results to establish confidence levels in predictions. 

• Schedule open-forum sessions to further engage interaction among all interested parties. 

• Maintain a public-domain accessible database of geometries, grids, and results. 

• Document workshop findings; disseminate this information through publications and presentations. 

DPW was envisioned to assess state-of-the-art CFD methods for prediction of aerodynamic flows of industrial 
interest. DPW was not planned as a verification and validation study. The primary tool of DPW is code-to-code 
comparison on a transonic transport aircraft problem of interest. Code-to-code comparison is used to evaluate how 
results differ, not as a verification study (see Oberkampf and Roy 1 for restrictions on using code-to-code comparison 
for code verification). DPW was intended as a measure of the state-of-the-art and was expected to identify some CFD 
topics that would require further investigation to improve the state-of-the-art. However, the code-to-code variation 
between the participants in the first DPW was substantially larger than expected. 2,3 The following four workshops were 
largely devoted to identifying and understanding the sources of variation with an expectation to reduce the variation 
between submitted results. To reduce the variation, the organizing committee added various elements of verification 
to the workshops. For example, the fifth and most recent workshop added solution verification of turbulence models 
on simple 2D configurations. 

The DPW series has consistently made approximations to modeling of the wind tunnel test that prevent the CFD 
predictions from being validated with the wind tunnel data. These approximations were made to increase participation 
and simplify the code-to-code comparisons. Wind tunnel data has been used as a guide to understand differences in 
CFD predictions. Several differences in CFD predictions were identified in the DPW Series that will require additional 
validation data to discriminate between results. 

This paper presents some observations from the series of five Drag Prediction Workshops that represent a tremen- 
dous amount of work from an international cast of collaborators. Each workshop has generated open discussions and 
a series of side studies to investigate various issues identified in the workshop. There is no attempt here to provide a 
thorough review of all of the DPW results and follow-on studies. However, many of the results and observations below 
are pulled from this rich set of studies. The DPW series has also helped to identify the need for validation data for flow 
separation. Additional observations from a recent validation experiment conducted at NASA Langley are provided to 
guide possible validation experiments. 


II. DPW Background 

DPW-I, held in June 2001 2,3 solicited CFD predictions of the lift, drag, and pitching moment for the DLR-F4 
subsonic transport wing-body configuration. The DLR-F4 wing-body configuration 4-6 was chosen due to public avail- 
ability of the geometry and experimental data from three wind tunnels. Test cases consisted of a single point solution at 
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a fixed value of Cl, calculation of a drag polar, and an optional calculation of drag rise at constant values of Cl- Grids 
were made available for participants, but DPW-I did not require a grid convergence study. A total of 38 solutions were 
submitted for the workshop from 18 participants using 13 different CFD codes. A summary of the results of DPW-I 
is given in Ref. 2, and a statistical analysis of the results is given in Ref. 3. The results of the code-to-code statistical 
analysis showed a disappointing 270 drag count spread in the fixed Cl data with a 100:1 confidence interval of more 
than ±50 drag counts. (Industry’s goal is ±1 drag count.) 

DPW-II, held in June 2003 7,8 focused on a grid refinement study and the prediction of installed pylon-nacelle drag 
increments. The DLR-F6 9 wing-body and wing-body-nacelle-pylon configurations were chosen for DPW-II since 
DLR and ONERA made data publicly available for this configuration. Test cases consisted of a single point solution 
at a fixed value of Cl and a drag polar for both the DLR-F6 wing-body and wing-body-pylon-nacelle configurations 
on coarse, medium, and fine grids. Optional test cases included a comparison of tripped and fully turbulent solutions 
and calculation of drag rise at fixed values of Cl. A total of 21 solutions were submitted for the workshop from 20 
participants using 18 different CFD codes. There were 16 solutions that calculated all three grid levels for both the 
DLR-F6 wing -body and wing-body -pylon-nacelle from 15 participants using 15 different CFD codes. A summary of 
the results is given in Ref. 7, and a statistical analysis of the results is given in Ref. 8. The DLR-F6 configuration had 
substantial areas of separation at the wing-body juncture and at the wing-pylon juncture. Additionally, there was a 
region of separation at the trailing edge of the wing. The code-to-code scatter for the wing-body configuration on the 
medium grid was significantly reduced compared to DPW-I, however, there was no significant change in code-to-code 
scatter with increasing grid density. 8, 10 

The panel session discussion at the conclusion of DPW-II identified three suggestions for a third workshop: (1) the 
large regions of separation were a likely culprit for the lack of grid convergence; therefore cases should be chosen 
with minimal separation, (2) simpler cases were required to allow for better grid convergence studies and wider 
participation, and (3) blind studies were preferable where experimental data were not available a priori. Vassberg 
et al. 11 designed a side-of-body fairing for DLR-F6 to produce attached flow in the wing-body juncture. The fairing 
was designated the FX2B, and the configuration was referred to as the DLR-F6-FX2B. Additionally, two isolated 
wings, designated DPW-W1 and DPW-W2, were designed 12 to be a simple geometry with DPW-W2 a single point 
optimization of DPW- W 1 . 

DPW-III, held in June 2006 10, 12 focused on grid convergence studies and predicted increments for the DLR-F6 
wing -body with and without the FX2B side-of-body fairing. The Reynolds number was increased from 3 to 5 million 
to minimize the trailing edge separation. Participants submitted solutions for DPW-III before the data for the FX2B 
fairing were collected in an experimental program at DLR and NASA. 13 Additional test cases included a drag polar 
at fixed Mach number and an optional Reynolds number scaling study. An optional grid convergence study included 
four grid levels for the DPW-W 1 and DPW-W2 isolated wing cases and a drag polar at a fixed Mach number where the 
angle-of-attack was specified rather than specifying a fixed Cl- A summary of the DPW-III results is given in Ref. 12, 
and a statistical analysis is given in Ref. 10. The code-to-code scatter for the DLR-F6 wing-body case was essentially 
the same for the separated (DLR-F6 no fairing) and attached (DLR-F6 with FX2B fairing) flow conditions. 10 The 
code-to-code scatter for the wing alone cases was noticeably lower than the wing-body cases, especially on coarse 
grids, when the variation due to lift was accounted for in the drag (through variation in lift-to-drag ratio and idealized 
profile drag coefficient). 10 

DPW-IV, held in June 2009 14,15 focused on prediction of both absolute and differential drag levels for wing-body 
and wing-body-horizontal tail configurations that are representative of transonic transport aircraft. Cases included 
a grid convergence study and a downwash study including prediction of trimmed drag on the NASA Common Re- 
search Model (CRM) wing-body-horizontal tail configuration. Optional cases included a Mach sweep study at fixed 
Cl and a Reynolds number study. The CRM was a new wing-body-horizontal tail configuration, with and with- 
out nacelle-pylons, developed by the NASA Subsonic Fixed Wing (SFW) Aerodynamics Technical Working Group 
(TWG) in collaboration with the DPW Organizing Committee. 16 The CRM is representative of a contemporary high- 
performance transonic commercial transport. DPW-IV was a blind test. The workshop was held in June 2009 and the 
experimental data were collected in the NASA Langley National Transonic Facility in January-February 2010 and in 
the NASA Ames 1 1-ft wind tunnel during March-April 2010. 17, 18 A summary of the results from DPW-IV is provided 
in Ref. 14, and a statistical analysis of the results is given in Ref. 15. 

The code-to-code scatter for the total drag, normalized by the total drag to account for different drag levels between 
cases, was virtually the same for DPW-II, DPW-III, and DPW-IV. The code-to-code variation of the forces and pitching 
moment, normalized by the appropriate force or pitching moment, was substantially larger on the horizontal tail 
component than on the wing or fuselage component. However, the scatter in the drag showed some reduction with 
increasing grid resolution. 15 
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The panel discussion at the conclusion of DPW-IV recommended continuing the workshop series with continuing 
focus on the CRM and a grid convergence study using a common set of grids for all CFD codes. DPW-V, held in June 
2012, focused on force and moment predictions for the CRM wing -body configuration, including a grid refinement 
study and an optional buffet study. The grid convergence study used a common grid sequence derived from a five 
block multiblock structured grid generated by Vassberg. 19 He generated six grid levels ranging from approximately 
640 thousand grid points on the coarsest grid to over 138 million grid points on the finest mesh. This multi-block 
grid was converted to overset, unstructured hexahedral, unstructured prisms, and unstructured hybrid grids by using 
the exact same set of grid points. The grid convergence study required at least four of the six grid levels. Optionally, 
participants could include solutions on grids they developed. The results on this common grid sequence showed a 
clear reduction in variation of forces and moments 20 over previous workshops. The optional wing-body buffet study at 
fixed Mach used the medium grid from the common grid sequence at seven angles of attack. Some solutions exhibited 
a large side of body separation bubble not observed in the wind tunnel results at off-design or high angles of attack. 
An optional third case was a turbulence model verification case. This verification case identified some turbulence 
model implementation issues and a few outliers in the submissions. A summary of the results from DPW-V is given 
in Ref. 21, and a statistical analysis of the results is given in Ref. 20. 

III. Observations from DPW 

The five Drag Prediction Workshops held to date provide a wealth of data on the state of CFD prediction of forces 
and moments for transonic transports. The workshops consist of participants presenting their results, summaries 
where all submitted results are plotted together, and extensive discussions. After the workshops, participants are 
provided an opportunity to submit additional results or to make corrections to results that were identified during the 
workshop. The process of a group of people from multiple institutions solving the same problem and plotting all of 
the results together is very powerful. The results at workshops were shared with the community at large through the 
DPW website 22 and special DPW-focused sessions organized at a conference typically six months or a year after the 
workshop. Additionally, many authors identified additional topics to study based on results of the workshop. 

First, a little background on grid convergence and the asymptotic range is important. The test cases used in DPW-I 
through DPW-V are designed to compare results of different CFD predictions. To keep the workshops as general and 
open to as many participants both in terms of case complexity and computational cost, some simplifying assumptions 
are made. For example, cases are typically run fully turbulent rather than matching the experimentally specified 
transition location and calculations are specified as configurations in free air rather than modeling the wind tunnel 
and mounting system. The following section addresses these physical model approximations more fully. The result 
of these approximations is that the wind tunnel data is not necessarily the “correct” result that the prediction should 
match. Rather, the prediction should have an offset from the wind tunnel results that represents corrections for each 
of the effects that are not modeled. We therefore look at how the solutions from the CFD predictions compare to one 
another — an N-version test. Hemsch 3 used a statistical approach that treated different computations as a collective 
and used N-version testing to investigate the submissions. No individual result is considered right or best. The 
statistical approach identified which solutions differed from the collective and was useful in identifying sources of 
variation. 3,8,10,15,20 Individual solutions could differ from the collective for multiple reasons. Different turbulence 
model, different numerical scheme, and errors are some of the sources of variation that were identified. Note, however, 
that understanding the reason for the variation is preferred over simply marking a solution as an outlier and ignoring it. 
Additionally, the mean of the solutions is occasionally used to investigate trends. However, it is not necessarily a better 
estimate of the “true” solution as it is merely an average of other solutions and has many coupled sources of variation. 
These studies provided a uniform method of evaluating trends in predictions over the course of the workshops. 

Another tool that has been used since DPW-II is the grid convergence study. The variation between different 
CFD predictions has components due to the grid and numerical approximations (discretized equations, grid), physical 
modeling (e.g., turbulence model), and errors (i.e., approximations that affect the solution in unintended ways or 
that aren’t what the author intended). One of the goals of the workshop series is to identify these different sources 
of variation, understand their effect, and recommend sources that should be further studied to improve predictive 
capability. The grid convergence study provides a method to study the variation due to the grids and numerical 
approximations. 

For a consistent numerical approximation of design order, p, the error will be reduced proportional to h p as the grid 
is refined, where h is a measure of the grid size. Salas 23,24 has shown that the error is linear with h p only if the grid 
is refined uniformly in all coordinate directions, e.g., if the grid spacing is halved in the x coordinate, it must also be 
halved in the y nd z coordinates. Other factors such as low-order boundary conditions can also contribute to possible 
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non-linearity in the error. A grid convergence study reduces the numerical error thereby reducing variation between 
CFD predictions due to grids and numerical approximation to isolate physical modeling and error differences. If the 
discretization error is reduced by the aforementioned rate as the grid is uniformly refined in all coordinate directions, 
the grid is said to be in the asymptotic range for that set of discretized equations. 

Baker 25 provides a physical interpretation of the asymptotic range as: all relevant flow features are captured to 
some degree on the target grid and further grid refinement only serves to sharpen their resolution. For Navier-Stokes 
computations this includes all shockwaves, expansions, vortex structures, boundary layers, wakes, and regions of flow 
separation. He further notes that the property of being in the asymptotic range is a function not only of the grid, but 
also of the set of discretized equations. 

To compare the predictions of two different physical models, e.g., the Spalart-Allmaras and the Reynolds stress 
transport turbulence models, reducing the numerical error below the differences in the models becomes critical. Oth- 
erwise the differences are a non-linear interaction of the turbulence model and the numerical discretization errors and 
can be misleading when trying to draw conclusions about physical model performance. 

An example of this can be seen in an axisymmetric example from Turgeon et al. 26 Figure 1 presents the length of 
the recirculation zone of the turbulent flow in a 30° axisymmetric diffuser obtained using an adaptive grid procedure 
with various numerical discretization schemes of the momentum and turbulence equations. All solutions are from 
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Figure 1: Recirculation length for 30° axisymmetric diffuser (from Turgeon et al. 26 ). 

the same CFD code with the same turbulence model. The variation between the solutions can be estimated using the 
range, which is clearly large on the coarse grids. As the grid is refined, the variation between the different numerical 
discretizations reduces until it eventually becomes zero on the finest grid where all of the discretizations give the same 
prediction of the recirculation length even though some of the discretizations completely miss the recirculation zone 
on the coarsest grids. The coarsest grid is not in the asymptotic range for the Galerkin/Galerkin, SUPG/CSU, and 
CSU/Galerkin discretizations since they predicted no recirculation. Since this is an adaptive grid sequence and not a 
uniform refinement, we don’t expect to see the error necessarily decay as h p . All discretizations converge to the same 
value of the recirculation length and the predictions are verified. 27 Only once the grid is converged can we accurately 
estimate the difference in the physical approximation. 

The following sections summarize some of the key observations from the Drag Prediction Workshops. These are 
collected into three categories: grids and numerical approximations, physical modeling, and open forums. 

A. Grids and Numerical Approximations 

After three workshops, the organizing committee recognized that a recurring theme of the workshop series was the 
importance of grid resolution and grid quality. Mavriplis et al. 28 provide an excellent review of grid issues in the 
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context of DPW-III results. 

Two different studies of the DLR-F6 used in DPW-II demonstrate that the differences due to the grids is larger than 
differences due to physical modeling. Laflin et al. 7 noted that the series of coarse, medium, and fine grids used by the 
participants were of insufficient density to obtain asymptotic convergence of the solution. In addition to grid size, the 
four cases used in DPW-II also provided a means to evaluate the effect of boundary layer transition specification (fully 
turbulent versus fixed transition location), turbulence model, and grid type on the range of predicted drag data, but it 
was shown that, “[n]one of these appear to have as significant an effect as grid size.” 7 

Mavriplis 29 performed a grid convergence and sensitivity study for the DLR-F6 wing-body used in DPW-II. Fig- 
ure 2 shows the lift and drag coefficients calculated at a subcritical (M = 0.30) and a transonic (M = 0.75) condition 
at zero incidence. The calculations were performed using two different levels of numerical dissipation. A grid conver- 
gence study was performed on a self-similar grid sequence consisting 1 million, 3 million, 9 million, and 72 million 
grid points. The drag values appear to be converging asymptotically although the lift values still show a small oscil- 
lation in convergence on this grid sequence. However, when the same cases are computed on a 65 million grid point 
grid that was generated using a similar resolution but with a different topology, dramatically different results are ob- 
tained. The effect of the two different grid topologies is substantially stronger than the level of numerical dissipation 
and swamps the effects due to two other modeling considerations studied: the turbulence model distance function 
calculation and the thin-layer Navier-Stokes assumption. 

Baker 30 applied the method developed in Baker 25 to evaluate whether results on the grid sequences used in DPW-II 
were in the asymptotic range. He identified nine of 23 submissions for the DLR-F6 wing-body that might be in the 
asymptotic range but only one of eight candidate submissions for the wing-body-nacelle-pylon. The remaining 21 
of 31 submissions were clearly not in the asymptotic range. He identified grid deficiencies and insufficient iterative 
convergence, especially for the large regions of separation on the wing-body-nacelle-pylon, as potential reasons that 
submissions were not in the asymptotic range. None of the sequences that were candidates for being in the asymptotic 
range had a calculated order of accuracy that matched the design order of accuracy of the numerical approximation. 
The fact that the calculated order of accuracy did not match the design order is a potential indicator that the grids did 
not meet the uniform refinement criteria. In a workshop setting, ensuring that all solutions have been converged to 
an acceptable level to perform an “order-analysis” is difficult. Baker 30 also noted that the number of significant digits 
used in reporting measures of interest was insufficient for performing the order analysis. He recommended requiring 
additional significant digits in future workshops. 

Eliasson et al. 31 examined the effect of grids on the side of body separation for the DPW-II DLR-F6 wing -body 
configuration. Calculations were made with the Edge solver with the same turbulence model (k-u EARSM) and the 
same numerical parameters; the only change was the grid families. Figure 3 shows a cut in the two grids at 15% 
wing span while Figure 4 shows the surface skin friction pattern at the wing-body junction for the two different 
unstructured grid families for three grid levels. A side of body separation bubble is visible for both the DLR grids and 
the ANSYS® grids. The separation on the ANSYS® grid is very small and weak. The separation on the DLR grid is 
a relatively extensive separation. The size of separation from the wind tunnel experiment appears to be between these 
two predictions. The separation region obtained with the DLR grids showed a better grid convergence in terms of its 
size and shape. The prediction of the separation region is clearly strongly influenced by the grid family; and so, from 
just these two solutions, determining which (if either) is the grid converged solution for the turbulence model is simply 
not possible. 

Submissions for the DPW-III DLR-F6 wing-body configuration with and without FX2B fairing unanimously 
agreed that the flow was well attached for the FX2B configuration but had large variation in the predicted size of 
the separation bubble for the DLR-F6 without the fairing. Vassberg et al. 12 noted that the level to which the uniform 
refinement criterion of the grid families was achieved varied across the grids. An objective of DPW-III was to test 
a hypothesis that pockets of flow separation can be a root cause of poor grid-convergence characteristics. Although 
it appears that pockets of flow separation did adversely affect the grid-convergence trends of many of the CFD data 
blocks provided by participants, it did not seem to cause issues with others. The adverse grid convergence trends are 
possibly an indicator of poor resolution, poor grid quality of some flow regions, and/or non-uniformly refined grid 
families. 

Sclafani et al. 32 demonstrated that the OVERFLOW code using structured overset grids achieved asymptotic 
convergence for the DPW-III DLR-F6 wing-body with FX2B fairing configuration and for the DPW-III DPW-W1/W2 
wing-alone cases. Asymptotic grid convergence was demonstrated on the medium/fine/extra fine sequence of grids 
for both the FX2B and the wing-alone configurations but the coarse grid was not in the asymptotic range for either 
configuration. However, the DLR-F6 wing-body without the FX2B fairing had a large side of body separation region 
and did not exhibit asymptotic convergence — the side of body separation zone grew with increasing grid resolution. 
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(a) Computed lift. 



(b) Computed drag. 

Figure 2: Comparison of computed lift and drag coefficients for the DLR-F6 wing -body configuration versus number 
of grid points to the —2/3 power for subsonic (M = 0.3) and transonic (M = 0.75) conditions at zero incidence 
on a sequence of uniformly refined grids and an alternate grid of 65 million points using the NSU3D solver (from 
Mavriplis 29 ). 
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(a) ANSYS® medium grid. (b) DLR medium grid. 

Figure 3: Grid cuts at 15% wing span for the DPW-II DLR-F6 wing-body configuration (from Eliasson et al. 31 ). 



(a) ANSYS® coarse grid. (b) ANSYS® medium grid. (c) ANSYS® fine grid. 



(d) DLR coarse grid. (e) DLR medium grid. (f) DLR fine grid. 

Figure 4: Surface friction pattern at wing-body junction from the ANSYS® and DLR grids at 1.23° angle of attack 
(from Eliasson et al. 31 ). 
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Meanwhile, Tinoco et al. 33 and Murayama and Yamamoto 34 demonstrated relatively constant separation bubble size 
with increasing grid resolution on multiblock grids. Similarly, separation at the wing trailing edge varies from no 
separation predicted by Sclafani et al. 32 to an increase in trailing edge separation with increasing grid density by 
Mavriplis 35 on an unstructured mesh. See Mavriplis et al. 28 for further discussion of the variation in prediction of 
the flow separation regions. The DPW-III DLR-F6 results did not show a consistent trend: some work demonstrated 
no side of body separation, some work showed a constant separation bubble size with increasing grid resolution, and 
some work showed increasing separation bubble size with increasing grid resolution. Clearly the grid effects have not 
been completely resolved for this complex flow field and more work is required to resolve this issue. 

The statistical summary for DPW-V 20 compared the grid convergence for DPW-II, DPW-III, DPW-IV, and DPW- V. 
The coefficient of variation of the total drag coefficient is plotted in Figure 5. The coefficient of variation, C V9 is used 



NPTS ' 2/3 


Figure 5: Coefficient of variation of total drag coefficient for DPW-II DLR-F6 wing-body, DPW-III DLR-F6 wing- 
body and DLR-F6 with FX2B fairing, DPW-IV CRM wing-body-horizontal tail, and DPW-V CRM wing-body (from 
Morrison 20 ). 

to compare variations from populations with different means. Figure 5 shows that the variation for DPW-II DLR-F6 
wing-body, DPW-III DLR-F6 wing-body with and without FX2B fairing, and DPW-IV CRM wing-body-horizontal 
tail all have a similar level of variation. The variation for the DPW-III DLR-F6 wing-body solutions with and without 
the FX2B fairing are almost identical. The variation for the DPW-V CRM wing-body configuration has substantially 
lower variation than previous workshops. The dotted line plotted with the DPW-V CRM results is an estimate of the 
uncertainty in the variation due to the different number of results for each of the six grid levels. The reduced variation 
in DPW-V results is primarily due to the use of a common grid sequence. 19 

Through five workshops the grid has been a leading order effect on the solutions. Gridding guidelines were 
provided in an effort to use best practices and minimize variation due to grids. However, conformance to the gridding 
guidelines has been inconsistent and is very difficult to verify. For example, given a sequence of grids, how can we 
evaluate how uniform the refinement sequence is? Ollivier-Gooch 36 described an approach for analyzing whether 
two meshes are geometrically similar and therefore appropriate for a mesh refinement study. He calculated the size, 
anisotropy, and orientation of cells based on moments of the cells. He then projected these quantities from one mesh 
to another to enable comparison. The directions of the anisotropy were also compared. Ollivier-Gooch 36 examined 
a pair of meshes from DPW-III and showed that cells near the wing trailing edge and on the fuselage of the coarse 
mesh cells are somewhat smaller than a uniform refinement of those regions would produce. Overall there were more 
cells on the coarse mesh that were either slightly smaller or larger than would be expected from a uniform refinement. 
Tools such as this need further refinement and application to a wider range of grids to evaluate their effectiveness. 

The workshops showed that the resolution and quality of grids affect the solution. Is there an a priori method 
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that can be used to estimate the quality of a solution on a given grid, which would enable improved grids before 
performing the CFD calculation? Dannenhoffer 37 investigated a priori grid metrics by using a 2D inviscid supersonic 
diamond airfoil. His systematic study computed the flow with more than 50 structured meshes and correlated a set of 
30 grid quality metrics with seven measures of the solution error. A statistical analysis of the results showed that there 
was essentially no correlation of the a priori grid quality metrics with the measures of solution error. Dannenhoffer 37 
concluded, “[t]his indicates that a priori ‘grid quality’ metrics can at most provide ‘loose guidance’ in the screening 
of computational grids.” 

Cavallo and Feldman 38 extended the work of Ollivier-Gooch 36 to provide a more compact means of assessment 
and comparison of grid sequences and to allow solution pairs to be evaluated as a means of evaluating similarity and 
the asymptotic range. They evaluated the NASA Langley node-based tetrahedral mesh sequence from DPW-IV and 
observed that the mesh distributions were nearly identical between the coarse-medium pair and the medium-fine pair. 
This confirmed that this grid sequence forms a self-similar family of meshes. 

Vos et al. 39 calculated the drag on the DPW-IV CRM by using both a near-field drag integration (integrating 
surface pressures and shear stresses) and a far-field drag integration procedure. They used the NSMB CFD solver with 
the same turbulence model and numerical parameters on six different multiblock grid sequences. They demonstrated 
approximately a 20 drag count variation between the multiblock grids with the near-field drag integration and about a 7 
drag count variation between the solutions with the far-field drag integration. Using the far-field integration approach 
they were able to identify regions on several of the multiblock grids that accounted for an increase in error of the drag 
estimate: lack of grid clustering near the wing leading edge for the cfse-ra and Zeus grids; lack of grid clustering at 
the wing trailing edge for the cfse-ra, ANSYS®, and Zeus grids; and streamwise spacing that is too large on the wing 
for the ANSYS® grid. 

Morrison 20 compared the forces and moments for each component (wing, fuselage, horizontal tail) of the DPW-IV 
DLR-F6 wing-body-horizontal tail and the DPW-V CRM wing-body configurations. Figure 6 shows that the total drag 
variation on the horizontal tail is approximately an order of magnitude larger than the variation on the fuselage. Also, 



NPTS- 2 * 


Figure 6: Coefficient of variation of lift coefficient for DPW-IV CRM wing, fuselage, and horizontal tail components 
and DPW-V CRM wing and fuselage components (from Morrison 20 ). 

the variation on the fuselage is approximately an order of magnitude larger than the variation on the wing. This tells us 
that differences between the total drag predictions is influenced more by the horizontal tail and the fuselage and infers 
that the grid resolution and grid quality is better on the wing and worse on the fuselage and tail. The lack of grid quality 
for the tail could be the grid over the horizontal tail and/or the wing wake grid which influences the inflow condition 
to the tail. This variation is possibly due to the state of the art in unstructured grid generation that tends to generate 
O-type grids around wing and tail surfaces, which leaves very coarse grids in the wakes. For such a grid topology, an 
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extreme amount of uniform refinement is required to reach the asymptotic range of the wake. Improvements to grids on 
the horizontal tail and fuselage components are also required to reduce the variation. Mavriplis et al. 28 also identified 
larger errors on the fuselage than the wing by examining the slope of the forces as a function of the average grid 
spacing. Lee-Rausch et al. 40 show similar trends for the DLR-F6 wing-body by using adjoint-based grid adaptation 
shown in Figure 7. In the figure, any non-blue color is a request for smaller grid spacing. Note the inset cut through 
the wing showing the request for more grid points in the wake. 



Figure 7: Adjoint-based adaptation parameter for the final medium- adapted mesh solution on the wing/body, symmetry 
plane and mid- span plane (inset) (from Lee-Rausch et al. 40 ). 


Observations 

The following observations can be made related to grids for transonic aerodynamic flows similar to the configurations 
tested in the various Drag Prediction Workshops. 

1 . Grid convergence on complex 3D problems is very difficult and probably requires more grid resolution than has 
been achieved in the DPW series. 

2. Grid resolution requirements involving regions of separated flow are much more demanding and less well un- 
derstood than attached flow. 28 

3. Generation of consistent grid families in the presence of complex geometries is a non-trivial task. 28 

4. Grid effects are most likely the dominant source of uncertainty. 

(a) Perform grid convergence studies for the chosen problem. 

i. Use uniform refinement and measure compliance of grids using a method such as Ollivier-Gooch 36 or 
Cavallo and Feldman. 38 

ii. Calculate the power of the convergence series by using the approach of Baker. 25 

iii. Compare the calculated convergence rate to the design order of the scheme. Identify deficiencies in 
the grid sequence, etc. and refine until the measured order of accuracy equals the design order. 
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(b) Use gridding guidelines determined from best practices of expert practitioners. Refine the gridding guide- 
lines. 

5. For steady solutions, fully converge results to steady state and report the level of convergence. Converge at least 
four orders of magnitude but preferably to machine zero. For unsteady problems, use a temporal error controller. 

6. Report figures of merit to sufficient accuracy for both computational and experimental results. 

B. Physical Modeling 

Three approximations to the wind tunnel experiment have consistently been used in the DPW series. First, since many 
CFD codes did not include transition prediction capability or transition specification capability, all required cases were 
modeled as fully turbulent. Each turbulence model has its own numerical transition behavior and will reach a fully 
developed turbulent condition at a location that depends on the initial turbulence levels and numerical dissipation. 
DPW-II included an optional case to evaluate the effects of fixing transition at the location specified in the wind 
tunnel experiment. However, the individual fully turbulent results were never evaluated to identify where transition 
occurred. Additionally, sensitivity to the initial turbulence levels was never evaluated. For validation, transition should 
be modeled based on the validation experiment. 

Second, the wind tunnel model is flexible and deflects under load; but to reduce the workload on grid generation 
and the participants, a single geometric definition for the configurations was used. DPW-II and DPW-III developed a 
finite element model of the DLR-F6 wind tunnel model and used the estimated deflection of the wind tunnel model 
at the design condition. DPW-IV and DPW-V also had a finite element model developed for the CRM wind tunnel 
model and used the estimated deflection of the wind tunnel model at the design condition. The experimental study for 
the CRM included photogrammetric measurement of the wind tunnel model during the test. This deflection data was 
compared to the estimated shape, and helped to identify that the wind tunnel model shape was different from the shape 
that was used for CFD in the workshops. 41,42 

Rivers et al. 42 studied the effect of the as-built wind tunnel model, the aeroelastic deflection, and the mounting 
system. Figure 8 shows the experimental data from both the NTF and the Ames tunnel tests, the DPW-V participant 
data, calculations with the aeroelastically-deflected wind tunnel model shape, and calculations with the sting mounting 
system added to the aeroelastic geometry. Each of the corrections improves the correlation of the prediction with the 
experimental data. The lift and drag coefficients show substantially closer agreement than the moment coefficient, but 
all show marked improvements when these effects are modeled. Additional studies 43 have confirmed that correcting 
the CFD model shape reduces the offset between CFD predictions and the wind tunnel data. The differences between 
the computational model and the wind tunnel model geometry will increase away from the design condition as the 
wind tunnel model deflection depends on the load and the CFD model was chosen to remain fixed throughout the 
workshop results. 

Third, wind tunnel data is typically provided with one or more corrections applied to correct from wind tunnel 
conditions to a free air condition. These corrections may include wind tunnel wall corrections, buoyancy corrections, 
sting effects, etc. Rivers 42 demonstrated that including sting effects in calculations improves comparisons between 
the CFD predictions and wind tunnel data for the CRM, which did not include any sting effect corrections to the 
data. Pfeiffer 44 provided a review of DPW-II results and discussed the wind tunnel installation and sting effects. He 
identified a testing plan that would allow for experimental determination of sting effects. However, this testing has not 
been attempted on any of the results used in the Drag Prediction Workshops. 

These three approximations to the physical modeling of the wind tunnel test limit the comparisons that can be 
made with the experimental data, and one should expect the predictions to show an offset from the experimental data 
due to these approximations. These approximations prevent the computations from being validated with the wind 
tunnel data. 

Figure 9 shows the pitching moment curves for the CRM wing-body and wing-body-horizontal tail at three tail 
incidence settings from DPW-IV. The Cm curves break between 3° and 4° for both the wing-body and the wing-body- 
horizontal tail. Most of the predictions break to the left (unstable). However, some of the predictions break to the 
right (more stable). Vassberg et al. 14 hypothesize that the curves breaking unstable correspond to outboard wing stall, 
while those breaking more stable probably predict a rapid increase in the side of body separation bubble. The CRM 
wing geometry was designed with an aggressive upper surface pressure gradient at the outboard section of the wing, 
which would lead to trailing edge separation. 16 As such, the test case was well designed to create a case to differentiate 
between model predictions. Further details of the predictions would be required to correlate the separation zones with 
the moment behavior, and more experimental data is required to identify the physical cause of the pitching moment 
break. Currently there is not sufficient experimental data to identify which, if either, behavior is correct. 
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NTF Test 197, Run 92 
Ames Test 216, Run 77 
WBTO with new w ing shape 
WBTOss with new wing shape 
WBTOssa with new wing shape 
DPW IV Participant Data 



Figure 8: Comparison of NTF and Ames 11 -ft TWT experimental data with the DPW-IV CRM CFD data for the 
wing-body-horizontal tail, in = 0°, (WBTO) configuration along with the USM3D CFD data for the WBTOss (wing- 
body-horizontal tail with support system) and WBTOssa (wing-body-horizontal tail with support system and arc sector) 
configurations using the new wing shape, M = 0.85, Re = 5 x 10 6 (from Rivers et al. 42 ). 
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CRM Wing-Body & Wing-Body-Hor izontsl Tail 
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Figure 9: DPW-IV CRM pitching moment curves (from Vassberg et al. 14 ). 


DPW-V included a turbulence model verification study 21 designed to determine if a given turbulence model was 
implemented in a CFD code as intended. There were three 2D cases provided from the Turbulence Model Resource 
website. 45 The verification test identified a discrepancy in the calculation of the minimum distance to the wall on a 
2D bump case from at least one participant. This discrepancy affected the skin friction prediction of the SA model. A 
correction to the wall distance calculation was implemented, and the prediction then matched the expected result. Ad- 
ditionally, the particular version of the SST model made a difference in the skin friction results. These differences are 
almost impossible to identify in a complex problem where there could be many differences in modeling the problem 46 
and where obtaining a grid converged solution is very difficult. Therefore, verification studies on simpler problems 
are critical to identify differences in numerical implementation. Additionally, many participants reported few details 
of their turbulence model. The Turbulence Model Resource Website includes standardized nomenclature of variations 
of turbulence models. This nomenclature should be used to clearly identify which turbulence model(s) the participant 
is using. Additional details of numerical implementation should also be included as they can often affect the results. 

Murayama et al. 47 investigated the effects of turbulence modeling for the DPW-V CRM wing-body configuration 
buffet onset case. The SA turbulence model predicted slightly larger the side of body separation than the SST turbu- 
lence model on a series of coarse, medium, and fine multiblock grids. The SST model actually showed no separation 
on the coarse grid while the SA showed separation similar in size to the fine grid. Clearly the coarse grid is not in 
the asymptotic range for the SST model. The SA model using the QCR (Quadratic Constitutive Relationship), which 
replaces the linear stress-strain Boussinesq relationship, predicted no separation for the same case. Figure 10 shows 
the surface-restricted streamlines on the wing surface for the SA turbulence model with and without the QCR model. 
The shock induced separation at the mid-span area increases with the QCR. However, there is no experimental data 
available to validate that the details of the flow prediction are correct. 

For DPW-V, Levy et al. 21 identified significant variation in lift and pitching moment in the buffet onset solutions at 
each angle of attack that were driven largely by shock location and the amount of trailing edge separation. Figure 1 1 
shows that there is massive separation by 4° angle of attack, and the separation exhibits significantly different patterns 
between solutions. There is no discernible trend. Levy et al. 21 raised the question whether this chaotic situation at the 
high angles of attack might be physical as well as computational. Are the steady Reynolds-averaged Navier-Stokes 
equations appropriate for modeling this flow regime? 
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(a) SA turbulence model. (b) SA-QCR turbulence model. 

Figure 10: Comparison of surface-restricted streamlines on the DPW-V CRM wing-body configuration wing surface 
with the SA turbulence model and the SA turbulence model with the QCR modification. Calculations using the UPACS 
code on the L3 medium common multi-block grid at M = 0.85, Re = 5 x 10 6 , 4° angle of attack (from Murayama 
et al. 45 ). 


Extensive validation experiments with detailed flow physics measurements are required to address several of these 
physical modeling issues. These experiments need to be designed to isolate effects and measure data that will distin- 
guish between models. 

Observations 

The following observations can be made related to physical modeling for transonic aerodynamic flows similar to the 
configurations tested in the various Drag Prediction Workshops. 

1 . Flow separation is challenging to simulate. 

2. Perform code verification first. Use simple cases to verify implementation of physical models. 

3. Report details of the turbulence model, or use common nomenclature 45 when reporting which model. 

4. Provide a series of cases with increasing complexity. Realistic configurations are very complex with unexpected 
areas of separation, etc. where many effects are confounded. Identify simpler cases to isolate the effects. 

5. Collect experimental data to distinguish effects, e.g., transition location, separation, Reynolds number effects. 

6. Raw experimental data plus all recommended wind tunnel corrections should be provided to the participants to 
permit an independent assessment of experimental error. 30,44 

7. Measure the geometry of the wind tunnel model before, during, and after the test. 30 

8. If simplifying assumptions are required, provide a method of evaluating sensitivity to those assumptions, e.g., 
fully turbulent. 
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(b) 4° angle of attack. 

Figure 11: Trailing edge separation on CRM at 3° and 4° angle of attack from DPW-V (from Levy et al. 21 ). 
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C. Open Forum 


Each workshop had standard grids generated for multi-block, overset, and unstructured grids. Beginning with DPW- 
II, these grids are distributed via the Internet and the organizers maintained a web site with the grids available. The 
hope was that these grids would continue to be used by other researchers. Participants were always encouraged to 
develop their own grids in addition to using the mandatory grids. However, beginning with DPW-III, participants who 
used their own grids were required to make these grids available via the DPW web site for others to use. This allows 
participants to evaluate effects of grids on solutions. 

The decision to keep grids available has proven to be an excellent decision. The organizing committee has con- 
tinuously been impressed with the amount of work that participants and other researchers are willing to do both for 
the workshops and well after the workshop. Participants and other researchers across the world continue to download 
grids from the workshops and run additional cases. New papers using these grids and results from previous workshops 
show up at many conferences. Each of these studies add to the body of results and increases our understanding of CFD 
predictive capability. 

The workshops have proven to be a very rich environment. Extensive discussions have been held in each workshop 
to identify trends and problems. 

Observations 

Workshops provide an excellent mechanism to bring many people together to focus on a problem and to address issues 
in how the problem is solved that does not happen in individual paper based results. 

1 . Make all grids available to participants — this allows participants to run grids used with other solvers to evaluate 
the effect of the grids on the solutions. 

2. Make all results available for others to analyze. 

IV. Validation Experiments 

As mentioned in the previous section, improving physical models requires not only insights to enable model 
creation but also very carefully executed experiments to validate those physical models. In this section we explore the 
lessons learned during a recent validation experiment campaign for supersonic retropropulsion (SRP) 48-50 and review 
some of the recommendation for conducting such a validation experiment as espoused by Oberkampf et al. 1,5 1,52 

In 2010, SRP (firing jets into the oncoming supersonic flow as a means of deceleration) was identified as an es- 
sential technology for five of nine entry, decent, and landing architectures studied for humans Mars exploration. 53 
Due to the nascent state of the SRP technology and the lack of ground test facilities to simulate Martian atmospheric 
conditions, a campaign was launched to begin validating CFD as a tool for predicting SRP environments. The first 
step was to conduct a validation experiment of a simple SRP model in air with non-reacting jets because as Ref. 51 
describes, “the common practice of attempting to validate codes using published data obtained for some purpose un- 
related to code validation” was found to be unsatisfactory. Previous SRP experimental test campaigns were conducted 
in the 1960s and 1970s; but because they were not designed and conducted as CFD validation experiments, they were 
unusable for CFD validation. The most frequent shortcomings of these early experiments were the lack of input char- 
acterization, e.g., freestream temperature or jet plenum temperature, a lack of detailed geometry, and complete absense 
of rigorous uncertainty quantification. Refs. 48 and 49 document the collaborative SRP validation experiment design, 
and here we will only highlight some of the insights and issues uncovered and during that process. 

First and foremost was the mantra, “Metior totum. Interrogo totum .” (Measure everything. Question everything.) 
For example, due to concerns regarding heat conduction and possible acoustic resonance in the jet plenums, the 
SRP experiment not only had temperature and pressure measurements for the main jet plenum manifold but also 
measurements just upstream of the individual jet nozzles and all were high-frequency transducers. 

One of the first steps of the SRP experimental design was to size the model to minimize tunnel wall interference by 
running preliminary CFD computations that compared the model surface pressure effects between enclosed (tunnel) 
and open (ideal) domains. Meanwhile, the wind tunnel from the plenum through the diffuser was digitized in case the 
full flow path needed to be modeled. 

Validation experiments are expensive. Do not under sell the test campaign to get your foot in the door. Be explicit 
and upfront with cost estimates for the extensive test matrix necessary to obtain quantified experimental uncertainties — 
see Ref. 51. Also, set clear, prioritized goals to maintain the focus on CFD validation. During the SRP validation 
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experiment campaign, the CFD validation goal was often overshadowed by desires to explore more jet and vehicle 
configurations, to make the experiment more flight-relevant, or to cut costs. 

Another insight was to avoid physics that are not essential to the problem under study. Search for the simplest 
problem that still contains the desired physics. If you can minimize the extent of separation, unsteady flow, shocks, and 
turbulent flow, do so. For example, the jet plenum air was heated to avoid jet plume liquefaction because condensation 
physics was not the focus of the validation effort or a consideration in the final application with reacting jets. And 
because the plenum air was heated, Berry et al. 48 further determined that to avoid heat conduction effects (loss of 
jet total temperature), heat- soaking the model prior to taking data was essential to keep the modeling as simple as 
possible. Similarly, if your problem involves turbulent flow, use trips as necessary to ensure transition (and preferably 
trips that can be readily modeled if necessary). Also, take care to choose designs that will not tend to re-laminarize. 
For example, add a fuselage/wing root leading edge horn so the turbulent flow along the fuselage remains so as it 
transitions onto the wing leading edge. 

Strive for as much symmetry as possible and keep the geometry as simple as possible. One of the SRP con- 
figurations had a central jet, so the model was axisymmetric (and indeed one of the experimental runs revealed a 
predominantly axisymmetric jet interaction mode). The other jet configurations had 120-degree symmetries (three 
nozzles). Other simplifications include making the nozzles conical instead of bell-shaped and enclosing the after body 
of the model in a cylindrical housing. In retrospect, one mistake was not adding an additional fairing to simplify 
modeling the high pressure line and instrumentation cables exiting the back of the model along the sting. After seeing 
the model mounted in the tunnel, photographs were taken; and a rough numerical model was created and simulated to 
gauge the effects of these excrescences. The deltas between ideal (clean) and as-tested were noticeable but thankfully 
their effects were confined to the rear of the model and away from the primary jet interaction region at the front. 

Try to establish a hierarchy of comparisons, starting with the simplest/easiest first. For example: a) body forces 
and moments; b) control surface forces and moments; c) surface pressure distributions; d) surface heat flux and shear 
stress; e) flowfield distributions of pressure, temperature, and velocity components; and f) flowfield distributions of 
Reynolds stresses. For the SRP validation experiment, a flow-through balance was not available to acquire integrated 
forces and moments, but the experiment did have extensive surface pressure measurements, a host of model and 
plenum temperature measurements, and high-speed schlieren videos. 

CFD cases computed prior to the tunnel entry allowed the use of the Virtual Diagnostics Interface (ViDI) methodol- 
ogy 54-56 to display comparisons of pre-computed and live, experimentally-measured, surface pressures. This provided 
a rapid, in situ, check of pressure tap mapping and calibration; and also allowed rapid identification of bad data chan- 
nels and tunnel interference effects as the model was traversed throughout the test section to collect measurements in 
support of statistical measurement uncertainty quantification. For the initial smooth forebody, no-jet cases, laminar 
CFD runs were used to double check the experimental data reduction system. Upon entering a new tunnel, an error 
was found in the data reduction algorithms thanks to this synergy between CFD and experiment. 

Observations 

The lessons learned during the validation experiment described above, fully support the following guidelines for 
conducting validation experiments from Ref. 48 and should be used when conducting validation experiments for 
transonic aerodynamic flows similar to the configurations tested in the various Drag Prediction Workshops. 

1 . A validation experiment should be jointly designed by experimentalists, model developers, code developers, and 
code users closely together throughout the program, from inception to documentation, with complete candor 
about the strengths and weaknesses of each approach. 

2. A validation experiment should be designed to capture the essential physics of interest, and measure all relevant 
physical modeling data, initial and boundary conditions, and system excitation information required by the 
model. 

3. A validation experiment should strive to emphasize the inherent synergism that is attainable between computa- 
tional and experimental approaches. 

4. Although the experimental design should be developed cooperatively, independence must be maintained in 
obtaining the computational and experimental system response results. 

5. Experimental measurements should be made of a hierarchy of system response quantities, for example, from 
globally integrated quantities to local quantities. 
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6. The experimental design should be constructed to analyze and estimate the components of random (precision) 
and systematic (bias) experimental uncertainties. 

V. Concluding Remarks 

The past decade of AIAA Drag Prediction Workshops and a recent validation experiment were mined for lessons 
learned to improve future workshops and associated experiments. These lessons were grouped into four areas: grid 
and numerical approximations, physical modeling, open forums, and validation experiments. Within each area several 
case studies were provided and followed by a summary list of observations for each area. For the grid and numerical 
approximation area, techniques for creating proper grid families were paramount. In the physical modeling area, the 
need for focused, validation experiments was evident while the open forum category underscored the multiplying 
power of open, transparent, communication. Finally, the validation experiments section underscored the benefits of 
following the guidelines developed by Oberkampf et al. 1,48,51,52 

Overarching observations include: 1) due to simplifications in the workshop test cases, wind tunnel data are not 
necessarily the “correct” results that CFD should match, 2) an average of core CFD data are not necessarily a better 
estimate of the true solution as it is merely an average of other solutions and has many coupled sources of variation, 3) 
outlier solutions should be investigated and understood, and 4) the DPW series does not have the systematic build up 
and definition on both the computational and experimental side that is required for detailed verification and validation. 
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